Foreach, future, and parallel computing with R

Bryan Lewis

Science

lapply

lapply
replicate

lapply
replicate
Reduce

lapply
replicate
Reduce
Map

Map? Or is it map? hmmmm....

lapply → mclapply

Map → mcMap

clusterWhateverApply

"Real HPC programmers use MPI"

Whatever.

Steve Weston

Package authors, code writers

Decide which parts of their code can run in parallel.

Package authors, code writers

Decide which parts of their code can run in parallel.

Users

Decide how to run in parallel.

> foreach == Philosophy
[1] TRUE

Haskell

Raskell

Despite the for-loop syntax, foreach works kinda like

Reduce(Map(...))

foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j) }

foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j) } i j 1 1 3 2 2 4 Warning message: executing %dopar% sequentially: no parallel backend registered

library(doParallel) registerDoParallel() foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j) } i j 1 1 3 2 2 4

← THIS TIME, IN PARALLEL

The future package is not opinionated, you can do the same thing with any syntax!

for loops
lapply
Map

map
replicate
no loop at all
...

Magic

Scoped objects are auto-magically exported

k <- pi foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j, k) } i j k 1 1 3 3.141593 2 2 4 3.141593

Compose two foreach loops with the %:% operator

Or compose a foreach loop with a filter predicate using 'when'

foreach(i = 1:10, j = 1:10) %:% when(i < j) %dopar% { i + j }

Future works interoperably with foreach through doFuture, as do other parallel adapters

doParallel
doSNOW
doMC
doMPI
doRedis

And a new guide to writing adapters:

https://github.com/bwlewis/writing_foreach_adapters/

Foreach, future, and parallel computing with R

Package authors, code writers

Package authors, code writers

Users

> foreach == Philosophy
[1] TRUE

Haskell

Raskell

Despite the for-loop syntax, foreach works kinda like

The future package is not opinionated, you can do the same thing with any syntax!

Magic

Scoped objects are auto-magically exported

Compose two foreach loops with the %:% operator

Or compose a foreach loop with a filter predicate using 'when'

Future works interoperably with foreach through doFuture, as do other parallel adapters

And a new guide to writing adapters:

Don't mourn...

Don't mourn...

parallelize!

Foreach, future, and parallel computing with R

Package authors, code writers

Package authors, code writers

Users

> foreach == Philosophy [1] TRUE

Haskell

Raskell

Despite the for-loop syntax, foreach works kinda like

The future package is not opinionated, you can do the same thing with any syntax!

Magic

Scoped objects are auto-magically exported

Compose two foreach loops with the %:% operator

Or compose a foreach loop with a filter predicate using 'when'

Future works interoperably with foreach through doFuture, as do other parallel adapters

And a new guide to writing adapters:

Don't mourn...

Don't mourn...

parallelize!

> foreach == Philosophy
[1] TRUE